Measurement system having a cooperative robot and three-dimensional imager

ABSTRACT

A measurement system and a method of measuring an object is provided. The system includes a measurement platform having a planar surface. At least two optical sensors are coupled to the measurement platform that emit light in a plane and determines a distance to an object based on a reflection of the light. A linear rail is coupled to the measurement platform. A cooperative robot is coupled to move along the linear rail. A 3D measuring system is coupled to the end of the robot. A controller coupled to the at least two optical sensors, the robot, and the 3D measuring system, the controller changing the speed of the robot and the 3D measuring system to less than a threshold in response to a distance measured by at least one of the at least two optical sensors to a human operator being less than a first distance threshold.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.No. 62/548,577, filed Aug. 22, 2017, the entire disclosure of which isincorporated herein by reference.

BACKGROUND

The subject matter disclosed herein relates in general to a humancollaborative robot having a triangulation-type, three-dimensional (3D)imager device, also known as a triangulation scanner.

A 3D imager uses a triangulation method to measure the 3D coordinates ofpoints on an object. The 3D imager usually includes a projector thatprojects onto a surface of the object either a pattern of light in aline or a pattern of light covering an area. A camera is coupled to theprojector in a fixed relationship, for example, by attaching a cameraand the projector to a common frame. The light emitted from theprojector is reflected off of the object surface and detected by thecamera. Since the camera and projector are arranged in a fixedrelationship, the distance to the object may be determined usingtrigonometric principles. Compared to coordinate measurement devicesthat use tactile probes, triangulation systems provide advantages inquickly acquiring coordinate data over a large area. As used herein, theresulting collection of 3D coordinate values or data points of theobject being measured by the triangulation system is referred to aspoint cloud data or simply a point cloud.

Robotic devices have been widely used in manufacturing and otherenvironments to reduce costs and improve quality. Robotic devices arehard/rigid bodies that may move in a rapid and unpredictable manner. Toavoid unintended impact with human operators, a typical manufacturingcell includes a lock-out procedure whereby the robot device is disabledwhen human operators need to enter the area. By locking out the roboticdevice it is ensured that the risk of contact by a moving robot iseliminated.

One type of robotic device has been developed, referred to as ahuman-centric or a collaborative robot, which allows the robot and thehuman operator to work in close proximity to each other while minimizingthe risk of impact to the human operator. These collaborative robotshave been proposed and used in a variety of applications, includingmedical facilities, libraries and manufacturing assembly operations.Collaborative robots include sensors that allow them to monitor theirsurrounding area including the presence of humans. The robot'scontroller is programmed to receive these sensor inputs and predict therisk of impact with nearby humans. When a potential impact on a human isdetected, the robot takes mitigating actions (e.g. slowing down orchanging direction) to avoid contact. In manufacturing environments,these human-centric robots have found use in light assembly and smallpart manufacturing.

Standards, such as ISO/TS 15066 (2016) and IS 13849-1:2015 for example,have been propagated to define desired performance levels andarchitecture of sensing systems used with human-centric robots. Thesestandards define operations of the systems to reduce contact riskbetween an operators and the robotic system. Sensing systems fall undera performance level “d” and category 3 of these standards. At this levelof performance, the sensing system needs to have reliability for onetype of failure as occurring once every 100-1000 years.

Accordingly, while existing triangulation-based 3D imager devices andcollaborative robots are suitable for their intended purpose the needfor improvement remains, particularly in providing a system formeasuring objects using an imager device that cooperates with acollaborative robot.

BRIEF DESCRIPTION

According to an embodiment of the present invention, a measurementsystem is provided. The system includes a measurement platform having aplanar surface. At least two optical sensors are coupled to themeasurement platform, the optical sensors each having a light source, anoptical sensor, and a processor that causes light from the light sourceto be emitted in a plane and determines a distance to an object based ona reflection of the light, the at least two optical sensors beingarranged to detect a human operator in a 360 degree area about themeasurement platform. A linear rail is coupled to the measurementplatform. A cooperative robot is coupled to move along the linear rail.A three-dimensional (3D) measuring system is coupled to the end of thecooperative robot, the 3D measuring system comprising: an imager devicehaving a projector, a first camera and a second camera arranged in apredetermined geometric relationship, the first camera and second cameraeach having a photosensitive array, the projector projecting a patternof light that includes at least one element; one or more firstprocessors operably coupled to the display, the projector, the firstcamera and the second camera, wherein the one or more first processorsare responsive to executable computer instructions for determining adistance to the at least one element; and a controller operably coupledto the at least two optical sensors, the cooperative robot, and the 3Dmeasuring system, the controller having one or more second processorsthat are responsive to executable instructions for changing the speed ofthe robot and the 3D measuring system to be less than a threshold inresponse to a distance measured by at least one of the at least twooptical sensors to a human operator being less than a first distancethreshold.

According to an embodiment of the present invention, a method ofmeasuring an object is provided. The method includes placing an objecton a measurement platform. A cooperative robot is moved at a first speedalong a rail coupled to the measurement platform. The object is scannedwith a three-dimensional (3D) measuring system coupled to an end of thecooperative robot, the 3D measuring system including an imager devicehaving a projector, a first camera and a second camera arranged in apredetermined geometric relationship, the first camera and second cameraeach having a photosensitive array, the projector projecting a patternof light that includes at least one element. A plane is scanned aboutthe measurement platform with at least one optical sensor. A humanoperator is detected, with the at least one optical sensor, at a firstdistance. The movement of the robot is changed to a second speed whenthe first distance is less than a first distance threshold, the secondspeed being less than the first speed.

These and other advantages and features will become more apparent fromthe following description taken in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter, which is regarded as the invention, is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The foregoing and other features and advantages ofthe invention are apparent from the following detailed description takenin conjunction with the accompanying drawings in which:

FIGS. 1-5 are views of measurement system according to an embodiment;

FIG. 6 is a perspective view of a 3D imager according to an embodiment;

FIG. 7 is a schematic illustration of the principle of operation of atriangulation scanner having a camera and a projector according to anembodiment;

FIG. 8 is a schematic illustration of the principle of operation of atriangulation scanner having two cameras and one projector according toan embodiment;

FIG. 9 is a perspective view of a scanner having two cameras and oneprojector arranged in a triangle for 3D measurement according to anembodiment;

FIGS. 10 and 11 are schematic illustrations of the principle ofoperation of the scanner of FIG. 9;

FIGS. 12-14 are top view schematic illustrations of measurement systemsillustrating the monitoring of areas around the measurement system,according to embodiments;

FIG. 15 is a flow diagram illustrating a method of operating themeasurement system of FIG. 1;

FIG. 16 is a schematic illustration of a safety zone about themeasurement system, according to embodiments;

FIGS. 17-20 are schematic illustrations of the operating zones of a 3Dimager, according to embodiments;

FIG. 21 is a schematic illustration of a 3D imager with spherical zones;and

FIG. 22 and FIG. 23 are perspective views of a measurement systemaccording to another embodiment.

The detailed description explains embodiments of the invention, togetherwith advantages and features, by way of example with reference to thedrawings.

DETAILED DESCRIPTION

Embodiments disclosed herein provide advantages in allowing automatedmeasurement or inspection of a part using a collaborative robot with athree-dimensional (3D) measurement device. Further embodiments providefor a measurement system having a platform for holding the object to beinspected, a robot that complies with compliant robot standards, and a3D measurement device coupled to the robot where the 3D measurementdevice does not comply with compliant robot standards.

Referring now to FIGS. 1-5, a measurement system 20 is shown thatincludes a platform 22. The platform 22 is a generally planar surfacethat is thermally stable and suitable for holding workpieces during aninspection process. The platform 22 may include features 24, such asholes or threaded holes for example, that may be used with auxiliarydevices, such as hold downs for example, that secure the workpiece tothe platform 22. The platform 22 is elevated off of the floor by a framemember 26. The frame member 26 may include openings 28 to allow anapparatus, such as a forklift for example, to move the measurementsystem 20. In an embodiment, the frame member 26 may include a recess orslot 30. As will be discussed in more detail herein, in an embodiment, atwo-dimensional (2D) optical sensor 32, 34 may be disposed at some orall of the corners of the slot 30.

In the illustrated embodiment, the measurement system 20 includes araised portion 36 that extends along one longitudinal side of theplatform 22. Mounted to the top of the raised portion 36 is a railassembly 38. The rail assembly 38 includes a slide 40 and a carriage 42.The carriage 42 may be mounted to the rail 40 via bearings (not shown)that allow the carriage 42 to slide along the length of the rail 40. Inan embodiment, the carriage 42 may be moved along the slide 40 usingmotors and belts (not shown) as is known in the art. Disposed atopposing ends of the slide 40 are stop members 44, 46. As will bediscussed in more detail herein, the member 44 includes a controller 48.

In the illustrated embodiment, an articulated robotic arm 50 is coupledto the carriage 42. The robotic arm 50 includes a plurality of armsegments 52, 54, 56 that are each rotatable about at least one axis atone end. In some cases, an arm segment may be able to rotate aboutmultiple axis (e.g. a hinge axis and a swivel axis). This allows therobotic arm 50 to articulate to place an end effector mounted to the endof segment 56 in a desired position and orientation with at leastsix-degrees of freedom. In the exemplary embodiment, an image scanner 60is mounted to the end of the robotic arm 50. The robotic arm 50 andimage scanner 60 are electrically coupled to the controller 48 via themember 44, such as via cabling (not shown) arranged internal to theslide 40 for example. In another embodiment, the cabling may be arrangedexternal to the slide 40, such as with a drag chain (not shown) thatextends between a base of the robotic arm 50 and the member 44.

In the exemplary embodiment, the robotic arm 50 is a human cooperativerobot, sometimes referred to as a human-centric robot. As used herein, acooperative robot 50 is a robotic device that is configured to operateautonomously or semi-autonomously in close proximity to a humanoperator. As used herein, the phrase “close proximity” means that thecooperative robot and the operator are positioned such that portions ofthe cooperative robot 50 may move within areas that overlap with thehuman operator during operations. As discussed herein, the controller 48is configured to alter the speed or movement of the cooperative robot 50to either avoid contact or reduce the impact on the human operator inthe event of contact. In an embodiment, the cooperative robot isconfigured to have a velocity at the point of contact of less than orequal to 255 millimeters/second, a maximum dynamic power of less than orequal to 80 Watts, or a maximum static force of less than or equal to150 Newtons. The sensors 32, 34 transmit a position signal that allowsthe determination of the position of the human operator. In oneembodiment, the cooperative robot 50 may incorporate the characteristicsfor inherent safety described in the journal article “A New ActuationApproach for Human-centric Robot Design” by Zinn et al. (Int. J ofRobotics Research, Vol. 23, No. 4-5, April-May 2004, pp. 379-398), thecontent of which is incorporated herein by reference. In anotherembodiment, the human-centric robot may include the characteristicsdescribed in journal article “Safety Evaluation of Physical Human-RobotInteraction via Crash Testing” by Haddadin et al. (Pro. of Robotics:Science and Systems III, June 2007), the content of which isincorporated herein by reference. In another embodiment, thehuman-centric robot may comply with ISO Standard ISO/DTS 15066, ISO/TR13482:2014 or ISO 10218 for example, the contents of which areincorporated by reference herein.

In an embodiment, the controller 48 is arranged within the stop member44. In an embodiment, the system 20 operation is controlled bycontroller 48. Controller 48 is a suitable electronic device capable ofaccepting data and instructions, executing the instructions to processthe data, presenting the results, and initiating and executing controlmethods. Controller 48 may accept instructions through user interface,or through other means such as but not limited to electronic data card,voice activation means, manually-operable selection and control means,radiated wavelength and electronic or electrical transfer. Therefore,controller 48 can be a microprocessor, microcomputer, an ASIC(application specific integrated circuit), a reduced instruction setcomputer, a digital computer, a supercomputer, a solid-state computer, asingle-board computer, a computer network, a desktop computer, a laptopcomputer, or a hybrid of any of the foregoing.

Controller 48 is capable of converting the analog voltage or currentlevel provided by sensors in the robotic arm 50, slide 40, carriage 42,or 3D imager into a digital signal indicative of the position of therobotic arm 50 or the 3D imager 60. Alternatively, sensors in therobotic arm 50, slide 40, carriage 42, or 3D imager 60 may be configuredto provide a digital signal to controller 48, or an analog-to-digital(A/D) converter (not shown) maybe coupled between sensors and controller48 to convert the analog signal provided by the sensors into a digitalsignal for processing by controller 48. Controller 48 uses the digitalsignals act as input to various processes for controlling the system 20.

As discussed herein controller 48 is operably coupled with one or morecomponents of system 20 such as the robotic arm 50 and the 3D imager 60by data transmission media. Data transmission media may include, but isnot limited to, twisted pair wiring, coaxial cable, and fiber opticcable. Data transmission media may also include, but is not limited to,wireless, radio and infrared signal transmission systems.

In general, controller 48 accepts data from sensors and is given certaininstructions for the purpose of comparing the data from sensors topredetermined operational parameters. Controller 48 provides operatingsignals to the robotic arm 50, the 3D imager 60, and the carriagemotors. Controller 48 also accepts data from the 3D imager 60, such as3D coordinate data for example. The controller 48 compares theoperational parameters to predetermined variances (e.g. position of the3D imager 60 relative to an inspection plan) and if the predeterminedvariance is exceeded, generates a signal that may be used to move thecarriage 42, robotic arm 60, or indicate an alarm to an operator on aconnected computer network. Additionally, the signal may initiate othercontrol methods that adapt the operation of the system 20 such aschanging the speed of the robotic arm 50 and 3D imager 60. For example,as will be discussed in more detail herein, if sensors 32, 34 detect thepresence of a person, the controller 48 may determine whether the personis within a predetermined area or zone about the system 20 based onparameters such as distance from the person to a side of system 20, thedirection of movement of the person and the speed of the person. Inresponse to determining the person is within the predetermined zone, thecontroller 48 may reduce the operating speed of the robotic arm 50 or 3Dimager 60 to be less than a predetermined speed, such as 255 mm/secondfor example.

In addition to being coupled to one or more components within system 20,the controller 48 may also be coupled to external computer networks suchas a local area network (LAN) and the Internet. LAN interconnects one ormore remote computers 50, which are configured to communicate withcontroller 48 using a well-known computer communications protocol suchas TCP/IP (Transmission Control Protocol/Internet(^) Protocol), RS-232,ModBus, and the like. Additional systems 20 may also be connected to LANwith the controllers 48 in each of these systems 20 being configured tosend and receive data to and from remote computers and other systems 20.LAN is connected to the Internet. This connection allows controller 48to communicate with one or more remote computers connected to theInternet.

In an embodiment, the controller 48 includes a processor coupled to arandom access memory (RAM) device, a non-volatile memory (NVM) device, aread-only memory (ROM) device, one or more input/output (I/O)controllers, and a LAN interface device via a data communications bus.

The LAN interface device provides for communication between controller48 and the LAN in a data communications protocol supported by LAN. TheROM device stores an application code, e.g., main functionalityfirmware, including initializing parameters, and boot code, for theprocessor. Application code also includes program instructions as shownin FIG. 15 for causing the processor to execute any operation controlmethods, including starting and stopping operation, changing operationalstates of robotic arm 50 or 3D imager 60, monitoring predeterminedoperating parameters such determining the location of human personsadjacent the system 20, and generation of alarms. The information to beexchanged remote computers and the controller 48 include but are notlimited to 3D coordinate data, operational state of 3D imager 60, andoperational state of the robotic arm 50.

The NVM device is any form of non-volatile memory such as an EPROM(Erasable Programmable Read Only Memory) chip, a disk drive, or thelike. Stored in NVM device are various operational parameters for theapplication code. The various operational parameters can be input to NVMdevice either locally, using a keypad, a connected computer (wired orwirelessly), a portable computing device (e.g. a table computer or acellular phone), a remote computer, or remotely via the Internet using aremote computer. It will be recognized that application code can bestored in the NVM device rather than the device.

Controller 48 includes operation control methods embodied in applicationcode shown in FIG. 15. These methods are embodied in computerinstructions written to be executed by processor, typically in the formof software. The software can be encoded in any language, including, butnot limited to, assembly language, VHDL (Verilog Hardware DescriptionLanguage), VHSIC HDL (Very High Speed IC Hardware Description Language),Fortran (formula translation), C, C++, C#, Objective-C, Visual C++,Java, ALGOL (algorithmic language), BASIC (beginners all-purposesymbolic instruction code), visual BASIC, ActiveX, HTML (HyperTextMarkup Language), Python, Ruby and any combination or derivative of atleast one of the foregoing. Additionally, an operator can use anexisting software application such as a spreadsheet or database andcorrelate various cells with the variables enumerated in the algorithms.Furthermore, the software can be independent of other software ordependent upon other software, such as in the form of integratedsoftware.

It should be appreciated that while controller 48 is described herein asbeing a single processing device, this is for exemplary purposes and theclaimed invention should not be so limited. In other embodiments, thecontroller 48 may include one or more processors. The one or moreprocessors may be co-located, such as in stop member 44 for example, ordistributed with some of the processors being remotely located from thestop member 44. Further, the controller 48 may be partially orcompletely separate from the stop member 44.

The system 20 may include other features to allow a human operator toeither interact with, or know the status of the system 20. In anembodiment, stop buttons 62 may be arranged on each side of the framemember 26. Further, lights 64 may be arranged on each side of the framemember 26. In an embodiment, the lights 4 are LED lights operable tochange color (e.g in response to a signal from the controller 48) basedon the operating state of the system 20. In an embodiment, the lights 64will be blue when in a waiting state (e.g. robotic arm 50 is locked inposition), amber when in a ready to operate state (e.g. the robotic arm50 is released for movement, but not yet moving) and red when in astopped/safety state.

Referring now to FIG. 6 is a perspective view of a 3D imager 60 is shownaccording to an embodiment. It includes a frame 602, a projector 604, afirst camera assembly 606, and a second camera assembly 608. The 3Dimager 60 may be the same as that described in commonly owned UnitedStates Patent Publication 2017/0054965A1, the contents of which areincorporated by reference herein.

FIG. 7 shows a structured light triangulation scanner 700 that projectsa pattern of light over an area on a surface 730. The scanner, which hasa frame of reference 760, includes a projector 710 and a camera 720. Theprojector 710 includes an illuminated projector pattern generator 712, aprojector lens 714, and a perspective center 718 through which a ray oflight 711 emerges. The ray of light 711 emerges from a corrected point716 having a corrected position on the pattern generator 712. In anembodiment, the point 716 has been corrected to account for aberrationsof the projector, including aberrations of the lens 714, in order tocause the ray to pass through the perspective center, therebysimplifying triangulation calculations.

The ray of light 711 intersects the surface 730 in a point 732, which isreflected (scattered) off the surface and sent through the camera lens724 to create a clear image of the pattern on the surface 730 on thesurface of a photosensitive array 722. The light from the point 732passes in a ray 721 through the camera perspective center 728 to form animage spot at the corrected point 726. The image spot is corrected inposition to correct for aberrations in the camera lens. A correspondenceis obtained between the point 726 on the photosensitive array 722 andthe point 716 on the illuminated projector pattern generator 712. Asexplained herein below, the correspondence may be obtained by using acoded or an uncoded (sequentially projected) pattern. Once thecorrespondence is known, the angles a and b in FIG. 9 may be determined.The baseline 740, which is a line segment drawn between the perspectivecenters 718 and 728, has a length C. Knowing the angles a, b and thelength C, all the angles and side lengths of the triangle 728-732-718may be determined. Digital image information is transmitted to aprocessor 750, which determines 3D coordinates of the surface 730. Theprocessor 750 may also instruct the illuminated pattern generator 712 togenerate an appropriate pattern. The processor 750 may be located withinthe scanner assembly, in controller 48, an external computer, or aremote server.

As used herein, the term “pose” refers to a combination of a positionand an orientation. In embodiment, the position and the orientation aredesired for the camera and the projector in a frame of reference of the3D imager 700. Since a position is characterized by three translationaldegrees of freedom (such as x, y, z) and an orientation is composed ofthree orientational degrees of freedom (such as roll, pitch, and yawangles), the term pose defines a total of six degrees of freedom. In atriangulation calculation, a relative pose of the camera and theprojector are desired within the frame of reference of the 3D imager. Asused herein, the term “relative pose” is used because the perspectivecenter of the camera or the projector can be located on an (arbitrary)origin of the 3D imager system; one direction (say the x axis) can beselected along the baseline; and one direction can be selectedperpendicular to the baseline and perpendicular to an optical axis. Inmost cases, a relative pose described by six degrees of freedom issufficient to perform the triangulation calculation. For example, theorigin of a 3D imager can be placed at the perspective center of thecamera. The baseline (between the camera perspective center and theprojector perspective center) may be selected to coincide with the xaxis of the 3D imager. The y axis may be selected perpendicular to thebaseline and the optical axis of the camera. Two additional angles ofrotation are used to fully define the orientation of the camera system.Three additional angles or rotation are used to fully define theorientation of the projector. In this embodiment, six degrees-of-freedomdefine the state of the 3D imager: one baseline, two camera angles, andthree projector angles. In other embodiment, other coordinaterepresentations are possible.

Referring now to FIG. 8 a structured light triangulation scanner 800 isshown having a projector 850, a first camera 810, and a second camera830. The projector creates a pattern of light on a pattern generatorplane 852, which it projects from a corrected point 853 on the patternthrough a perspective center 858 (point D) of the lens 854 onto anobject surface 870 at a point 872 (point F). The point 872 is imaged bythe first camera 810 by receiving a ray of light from the point 872through a perspective center 818 (point E) of a lens 814 onto thesurface of a photosensitive array 812 of the camera as a corrected point820. The point 820 is corrected in the read-out data by applying acorrection factor to remove the effects of lens aberrations. The point872 is likewise imaged by the second camera 830 by receiving a ray oflight from the point 872 through a perspective center 838 (point C) ofthe lens 834 onto the surface of a photosensitive array 832 of thesecond camera as a corrected point 835.

The inclusion of two cameras 810 and 830 in the system 800 providesadvantages over the device of FIG. 8 that includes a single camera. Oneadvantage is that each of the two cameras has a different view of thepoint 872 (point F). Because of this difference in viewpoints, it ispossible in some cases to see features that would otherwise beobscured—for example, seeing into a hole or behind a blockage. Inaddition, it is possible in the system 800 of FIG. 8 to perform threetriangulation calculations rather than a single triangulationcalculation, thereby improving measurement accuracy. A firsttriangulation calculation can be made between corresponding points inthe two cameras using the triangle CEF with the baseline B₃. A secondtriangulation calculation can be made based on corresponding points ofthe first camera and the projector using the triangle DEF with thebaseline B₂. A third triangulation calculation can be made based oncorresponding points of the second camera and the projector using thetriangle CDF with the baseline B₁. The optical axis of the first camera820 is 816, and the optical axis of the second camera 830 is 836.

FIG. 9 shows 3D imager 900 having two cameras 910, 930 and a projector950 arranged in a triangle A₁-A₂-A₃. In an embodiment, the 3D imager 900further includes a camera 990 that may be used to provide color(texture) information for incorporation into the 3D image. In addition,the camera 990 may be used to register multiple 3D images through theuse of videogrammetry.

This triangular arrangement provides additional information beyond thatavailable for two cameras and a projector arranged in a straight line asillustrated in FIG. 6 and FIG. 8. The additional information may beunderstood in reference to FIG. 10, which explain the concept ofepipolar constraints, and FIG. 11 that explains how epipolar constraintsare advantageously applied to the triangular arrangement of the 3Dimager 900. In FIG. 10, a 3D triangulation instrument 1040 includes adevice 1 and a device 2 on the left and right sides of FIG. 10,respectively. Device 1 and device 2 may be two cameras or device 1 anddevice 2 may be one camera and one projector. Each of the two devices,whether a camera or a projector, has a perspective center, O₁ and O₂,and a representative plane, 1030 or 1010. The perspective centers areseparated by a baseline distance B, which is the length of the line1002. Basically, the perspective centers O₁, O₂ are points through whichrays of light may be considered to travel, either to or from a point onan object. These rays of light either emerge from an illuminatedprojector pattern, such as the pattern on illuminated projector patterngenerator 712 of FIG. 7, or impinge on a photosensitive array, such asthe photosensitive array 722 of FIG. 7. As can be seen in FIG. 7, thelens 714 lies between the illuminated object point 732 and plane of theilluminated object projector pattern generator 712. Likewise, the lens724 lies between the illuminated object point 732 and the plane of thephotosensitive array 722, respectively. However, the pattern of thefront surface planes of devices 712 and 722 would be the same if theywere moved to appropriate positions opposite the lenses 714 and 724,respectively. This placement of the reference planes 1030, 1010 isapplied in FIG. 10, which shows the reference planes 1030, 1010 betweenthe object point and the perspective centers O₁, O₂.

In FIG. 10, for the reference plane 1030 angled toward the perspectivecenter O₂ and the reference plane 1010 angled toward the perspectivecenter O₁, a line 1002 drawn between the perspective centers O₁ and O₂crosses the planes 1030 and 1010 at the epipole points E₁, E₂,respectively. Consider a point U_(D) on the plane 1030. If device 1 is acamera, it is known that an object point that produces the point U_(D)on the image lies on the line 1038. The object point might be, forexample, one of the points V_(A), V_(B), V_(C), or V_(D). These fourobject points correspond to the points W_(A), W_(B), W_(C), W_(D),respectively, on the reference plane 1010 of device 2. This is truewhether device 2 is a camera or a projector. It is also true that thefour points lie on a straight line 1012 in the plane 1010. This line,which is the line of intersection of the reference plane 1010 with theplane of O₁-O₂-U_(D), is referred to as the epipolar line 1012. Itfollows that any epipolar line on the reference plane 1010 passesthrough the epipole E₂. Just as there is an epipolar line on thereference plane of device 2 for any point on the reference plane ofdevice 1, there is also an epipolar line 1034 on the reference plane ofdevice 1 for any point on the reference plane of device 2.

FIG. 11 illustrates the epipolar relationships for a 3D imager 1090corresponding to 3D imager 900 of FIG. 9 in which two cameras and oneprojector are arranged in a triangular pattern. In general, the device1, device 2, and device 3 may be any combination of cameras andprojectors as long as at least one of the devices is a camera. Each ofthe three devices 1191, 1192, 1193 has a perspective center O₁, O₂, O₃,respectively, and a reference plane 1160, 1170, and 1180, respectively.Each pair of devices has a pair of epipoles. Device 1 and device 2 haveepipoles E₁₂, E₂₁ on the planes 1160, 1170, respectively. Device 1 anddevice 3 have epipoles E₁₃, E₃₁, respectively on the planes 1160, 1180,respectively. Device 2 and device 3 have epipoles E₂₃, E₃₂ on the planes1170, 1180, respectively. In other words, each reference plane includestwo epipoles. The reference plane for device 1 includes epipoles E₁₂ andE₁₃. The reference plane for device 2 includes epipoles E₂₁ and E₂₃. Thereference plane for device 3 includes epipoles E₃₁ and E₃₂.

Consider the situation of FIG. 11 in which device 3 is a projector,device 1 is a first camera, and device 2 is a second camera. Supposethat a projection point P₃, a first image point P₁, and a second imagepoint P₂ are obtained in a measurement. These results can be checked forconsistency in the following way.

To check the consistency of the image point P₁, intersect the planeP₃-E₃₁-E₁₃ with the reference plane 1160 to obtain the epipolar line1164. Intersect the plane P₂-E₂₁-E₁₂ to obtain the epipolar line 1162.If the image point P₁ has been determined consistently, the observedimage point P₁ will lie on the intersection of the determined epipolarlines 1162 and 1164.

To check the consistency of the image point P₂, intersect the planeP₃-E₃₂-E₂₃ with the reference plane 1170 to obtain the epipolar line1174. Intersect the plane P₁-E₁₂-E₂₁ to obtain the epipolar line 1172.If the image point P₂ has been determined consistently, the observedimage point P₂ will lie on the intersection of the determined epipolarlines 1172 and 1174.

To check the consistency of the projection point P₃, intersect the planeP₂-E₂₃-E₃₂ with the reference plane 1180 to obtain the epipolar line1184. Intersect the plane P₁-E₁₃-E₃₁ to obtain the epipolar line 1182.If the projection point P₃ has been determined consistently, theprojection point P₃ will lie on the intersection of the determinedepipolar lines 1182 and 1184.

The redundancy of information provided by using a 3D imager 900 having atriangular arrangement of projector and cameras may be used to reducemeasurement time, to identify errors, and to automatically updatecompensation/calibration parameters.

One method of determining 3D coordinates is by performing sequentialmeasurements. An example of such a sequential measurement methoddescribed herein below is to project a sinusoidal measurement patternthree or more times, with the phase of the pattern shifted each time. Inan embodiment, such projections may be performed first with a coarsesinusoidal pattern, followed by a medium-resolution sinusoidal pattern,followed by a fine sinusoidal pattern. In this instance, the coarsesinusoidal pattern is used to obtain an approximate position of anobject point in space. The medium-resolution and fine patterns used toobtain increasingly accurate estimates of the 3D coordinates of theobject point in space. In an embodiment, redundant information providedby the triangular arrangement of the 3D imager 900 eliminates the stepof performing a coarse phase measurement. Instead, the informationprovided on the three reference planes 1160, 1170, and 1180 enables acoarse determination of object point position. One way to make thiscoarse determination is by iteratively solving for the position ofobject points based on an optimization procedure. For example, in onesuch procedure, a sum of squared residual errors is minimized to selectthe best-guess positions for the object points in space.

The triangular arrangement of 3D imager 1190 may also be used to helpidentify errors. For example, a projector 1193 in a 3D imager 900 mayproject a coded pattern onto an object in a single shot with a firstelement of the pattern having a projection point P₃. The first camera1191 (FIG. 11) may associate a first image point P₁ on the referenceplane 1160 with the first element. The second camera 1192 may associatethe first image point P₂ on the reference plane 1170 with the firstelement. The six epipolar lines may be generated from the three pointsP₁, P₂, and P₃ using the method described herein above. The intersectionof the epipolar lines lie on the corresponding points P₁, P₂, and P₃ forthe solution to be consistent. If the solution is not consistent,additional measurements of other actions may be advisable.

The triangular arrangement of the 3D imager 900 may also be used toautomatically update compensation/calibration parameters. Compensationparameters are numerical values stored in memory, for example, in thecontroller 48 or in another external computing unit. Such parameters mayinclude the relative positions and orientations of the cameras andprojector in the 3D imager.

The compensation parameters may relate to lens characteristics such aslens focal length and lens aberrations. They may also relate to changesin environmental conditions such as temperature. Sometimes the termcalibration is used in place of the term compensation. Oftencompensation procedures are performed by the manufacturer to obtaincompensation parameters for a 3D imager. In addition, compensationprocedures are often performed by a user. User compensation proceduresmay be performed when there are changes in environmental conditions suchas temperature. User compensation procedures may also be performed whenprojector or camera lenses are changed or after then instrument issubjected to a mechanical shock. Typically user compensations mayinclude imaging a collection of marks on a calibration plate.

It should be appreciated that the 3D imager 60, when mounted to therobotic arm 50 may be used to automatically follow a predeterminedinspection plan based on inspection data stored in the controller 48. Assuch, the inspection of a workpiece placed on the platform 24 may beperformed consistently, reliably and quickly. Saving both time and moneyfor the workpiece manufacturer as a result.

In some embodiments, while the robotic arm 50 is a cooperative robot andcomplies with the standards for working in close proximity to humanoperators, the end effectors mounted to the arm segment 56 are not. Oneof the standards of a cooperative robot is that any corner on the robotneeds to have a minimum radius of 5 mm-10 mm. As a result, when an endeffector, such as a 3D imager 60 is mounted to the cooperative roboticarm 50, the operation of the system 20 needs to be adjusted to avoidundesired contact between the robotic arm 50, the 3D imager 60 and thehuman operator.

In an embodiment, the system 20 includes two 2D optical sensors 32, 34.These sensors 32, 34 are located at opposite corners of the frame member26 at a predetermined distance from the floor or surface that the system20 is placed. In an embodiment, the 2D optical sensors are Hokuyo SafetyScanner UAM-05LP-T301 or Sick S300/microScan3 Core/S3000. The opticalsensors 32, 34 emit light in a plane and are operable to determine adistance to an object having a surface in the plane. As shown in FIG.12, the optical sensor 32 emits light in a plane 1200 about the system20 and the optical sensor 32 emits a light in a plane 1202 about thesystem 20. In an embodiment, the planes of light 1200, 1202 are at adistance of 200 mm above the floor.

Referring now to FIG. 13, an embodiment is shown of the system 20 with acooperative robotic arm 50 and a 3D imager 60 that has corners withradii less than 5 mm. In this embodiment, the system 20 has 2 modes ofoperation. In the first mode of operation, there are no human operatorsin proximity to the platform 22 and the robotic arm 50. In the firstmode of operation, the robotic arm 50 may be operated at speeds greatthan a predetermined speed, such as greater than 255 mm/second. In asecond mode of operation, the speed of the robotic arm 50 is adjustedbased on whether the operator is in a safety or first zone 1302 (FIG.16), or within an information or second zone 1300. The first zone 1302is defined by a space about the workpiece 1304 that neither the roboticarm 50 nor the 3D imager 60 will enter when the system 20 is operatingin the second mode (e.g. a human operator is present). In an embodiment,the first zone 1302 is defined as being equal to or greater than 100 mmfrom the side and top surfaces of the workpiece 1304. In one embodiment,the controller 48 verifies (either continually, periodically oraperiodically) that that speed is less than a threshold speed (e.g. lessthan 255 mm/second). In another embodiment, the controller 48 stops themovement of the robotic arm 50 when a human operator or a portion of thehuman operator is within the first zone 1302.

The second zone 1300 is determined by the controller 48 based at leastin part on the speed that the robotic arm 50 is operating. In anembodiment, the size of the second zone 1300 is based at least in parton the approach speed of the human operator (K), such as operator 1306for example, the stopping/run-down time of the robotic arm 50 andcarriage 42 (T_(M)), the response time of the system 20 (T_(S)) and areaching constant (C). The reaching constant is a variable to accountfor the operator 1306 extending their arm and hand. In an embodiment,the distance S from the edge of the frame member 26 is determined by thecontroller 48 by the following relationship:S=(K*(T _(M) +T _(S)))+C   (Equation 1)

In an embodiment, the approach speed K is fixed as being 1600 mm/second.In another embodiment, the approach speed K is calculated by thecontroller 48 based on signals from the sensors 32, 34.

For example, if the operator 1306 is moving at a speed of 1600 mm/s andthe system stopping/run-down time is 0.2 ms and the response time is 0.3ms, then the distance S=(1600 mm/s*(0.2 ms+0.3 ms))+1200 mm=2000 mm.

In one embodiment, the reaching constant C may be calculated based onthe height of the plane of light 1200, 1202 from the floor. It should beappreciated that the higher off the ground the less reaching distancepotential for the operator 1306. In this embodiment, the reachingconstant C is determined by the relationship:C=1200 mm−(0.4*H _(D))   (Equation 2)

where HD is the height of the plane of light 1200, 1202 above the floor.For example, where the planes 1200, 1202 are 200 mm above the floor, thereaching constant C=1200 mm−(0.4*200 mm), or 850 mm. Thus the distance Swould be 350 mm less than if the fixed constant of 1200 mm is used.

In the embodiment of FIG. 13, the second zone 1200 is a fixed distancefrom the sides of the frame member 26 about the perimeter system 20. Ifthe operator 1306 was to continue moving and enter the second zone 1300,the controller 48 would decrease the speed of the robotic arm 50 from anautomatic speed mode in full speed un collaborative mode to less than apredetermined threshold speed, such as 255 mm/s. In another embodiment,the system 20 stops the movement of the robotic arm 50 (e.g. 0 mm/s).

Referring now to FIG. 14, another embodiment of the system 20 is shownwhere the size of the second zone 1400 changes based on the position ofthe robotic arm 50. In this embodiment, the distance S is changed basedat least in part on the movement of the robotic arm 50. As the roboticarm 50 is moved in a direction, either along the slide 40, such as inthe direction indicated by arrow 1402. As the robotic arm 50 moves in adirection, the distance S along the edge in that direction extends tocreate a larger space in the area that the robotic arm 50 is movingtowards. Similarly, the distance S′ of the area the robotic arm 50 ismoving away from is reduced. In an embodiment, the system 20 maintains aminimum distance S′, such as 500 mm for example.

Referring now to FIG. 15, a method 1500 is shown for operating thesystem 20. The method 1500 starts in block 1502 and proceeds to block1504 where the 3D imager 60 and the collaborative robotic arm 50 isactivated in block 1506. It should be appreciated that the controller 48may be preprogramed with a desired inspection plan (e.g. measurements tobe made by the 3D imager 60) for the workpiece placed on the platform24. In block 1506 the robotic arm 50 moves the 3D imager 60 to thedesired locations to make measurements defined by the inspection plan.The method 1500 then proceeds to block 1508 where the distance S isdetermined to define the second zone. When no human operators arepresent or within the second zone, the system 20 operates in the firstoperating mode (high speed).

The method 1500 then proceeds to query block 1510 where it is determinedif a human operator is detected by the optical sensors 32, 34. If queryblock 1510 returns a negative, the method 1500 loops back to block 1506and the measurement of the workpiece continues. When the query block1510 returns a positive, the method 1500 proceeds to block 1512 isdetermined from the optical sensors 32, 34. In one embodiment, the speedand direction of movement of the human operator is also determined. Themethod 1500 then proceeds to query block 1514 where it is determined ifthe human operator is within the second zone. When the query block 1514returns a negative, the method 1500 loops back to block 1506 and themeasurements of the workpiece continue.

When the query block 1514 returns a positive, meaning the operator iswithin the second zone, the method 1500 then proceeds to query block1516. In query block 1516, it is determined if the human operator iswithin the first zone. If the query block 1516 returns a positive (e.g.human operator within zone one), the method 1500 proceeds to block 1518where the speed of the robotic arm 50 is verified to be less than thethreshold speed (e.g. less than 255 mm/second). In zone embodiment, themethod 1500 stops (e.g. speed of 0 mm/second) the robotic arm 50 whenthe operator is within the first zone. The method 1500 then loops backto block 1506 and the measurements of the work piece continue. When thequery block 1516 returns a negative (e.g. human operator in secondzone), the method 1500 proceeds to block 1520 where a reduction of thespeed of the robotic arm 50 is initiated. In one embodiment, the rate ofreduction is defined to reduce the speed of the robotic arm 50 to beless than or equal to a predetermined speed (e.g. 255 mm/second) by thetime the operator reaches the first zone. In an embodiment, the speedand direction of the human operator is determined.

FIGS. 17-19 show two versions 60A, 60B of the 3D imager 10. The 3Dimager 60B includes relatively wide FOV projector and camera lenses,while the 3D imager 60A includes relatively narrow FOV projector andcamera lenses. The FOVs of the wide-FOV cameras 606, 608 and projector604 (FIG. 6) of FIGS. 19-20 are cameras 606B, 60B and projector 604B,respectively. The FOVs of the narrow-FOV cameras 606, 608 and projector604 of FIGS. 17-18 are cameras 606A, 608A and projector 604A,respectively. As used herein, the standoff distance D of the 3D imager60B is the distance from the front of the projector lens 604B to thepoint of intersection of the optical axes of the camera lens assemblies606B, 608B, respectively, with the optical axis of the projector 604B.The standoff distance D_(A) of the 3D imager 60A is defined in the sameway. In the illustrated embodiment, the standoff distances D_(A), D_(B)for the 3D imagers 660A, 60B are different. In other embodiments, thestandoff distance D_(A) of the 3D imager 60A is the same as the standoffdistance D_(B) of the 3D imager 60B.

In an embodiment, the narrow-FOV camera lenses of 3D imager 60A may havelonger focal lengths than the wide-FOV camera lenses of 3D imager 60B ifthe photosensitive array is the same size in each case. In addition, asshown in FIGS. 19-20, the width of the measurement region 1700 issmaller than the width of the measurement region 1900. In addition, ifthe diameters of lens apertures are the same in each case, the depth(the depth of field (DOF)) of the measurement region 1700 is smallerthan the depth (DOF) of the measurement region 1900. In an embodiment,3D imagers 60 are available with different fields of view and differentimage sensor resolution and size.

It should be appreciated that it is undesirable for the 3D imager 60A,60B or any portion thereof from entering the first zone 1302 about theworkpiece 1304. As a result, the minimum thickness of a portion of theworkpiece 1304 may be different due to the difference in the size of themeasurement regions 1700, 1900. In some embodiments, depending on thesize of the workpiece, the first zone standoff distance a, the standoffdistance D, and the measurement region size 1700, 1900, there may be aminimum thickness of the workpiece 1304 that the 3D imager 60A, 60B canmeasure. For example, in the embodiment of FIGS. 17-18, for a workpiecehaving a maximum height of 50 cm, a first zone standoff distance of 100mm, a 3D imager standoff distance D_(A) of 90 mm. Due to the smallersize of the measurement region 1700, the minimum thickness of theworkpiece that may be measured is 5 mm.

In an embodiment, the system 20 performs an initial scan of theworkpiece 1304 to identify the bounds of the workpiece 1304. Based onthis initial scan, the system 20 determines whether the 3D scanner 60 iscapable of measuring to the platform 22. In an embodiment, the system 20transmits a signal to the operator to alert to notify them that thesystem 20 may not be able to measure at least some aspects of theworkpiece 1304.

In an embodiment, shown in FIG. 21, the system 20 creates an additionalzone 2100 about the 3D imager 60 to avoid a collision between the 3Dimager 60 and the workpiece 1304. In an embodiment, the system 20generates six spherical zones 2102 around the 3D imager 60. Thespherical zones 2102 generate an area around the 3D imager 60 whereobjects (such as workpiece 1304) should not enter. In other words, thesystem 20 assumes the 3D imager is larger in size than the actual 3Dimager 60. In one embodiment, the spherical zones 2102 are placed aroundor centered on the cameras 606, 608, projector 604, the rear corners andthe rear-center of the 3D imager 60.

Referring now to FIG. 22 and FIG. 23, another embodiment of themeasurement system 20 is shown. This embodiment is similar to that shownin FIGS. 1-5, except that the platform 22 is elevated off the floor by aframe member 2200. In this embodiment, the frame 2200 includes a slot2202 that is position adjacent the floor that the system 20 rests.Similar to slot 30 (FIG. 1), the optical sensors 32, 34 are disposed inat least two corners of the slot 2202. In an embodiment, the opticalsensors are disposed at opposing corners. The optical sensors 32, 34define a plane (e.g. plane 1200, 202—FIG. 12) that is generally parallelto, and adjacent with, the floor.

In an embodiment, the system 20 further includes a connector member2300. The connector member 2300 provides a connection point forelectrical power and data to the system 20. In an embodiment, theconnector member 2300 is disposed within the slot 2202 and placed toavoid having cables interfere with the travel or operation of therobotic arm 50.

While the invention has been described in detail in connection with onlya limited number of embodiments, it should be readily understood thatthe invention is not limited to such disclosed embodiments. Rather, theinvention can be modified to incorporate any number of variations,alterations, substitutions or equivalent arrangements not heretoforedescribed, but which are commensurate with the spirit and scope of theinvention. Additionally, while various embodiments of the invention havebeen described, it is to be understood that aspects of the invention mayinclude only some of the described embodiments. Accordingly, theinvention is not to be seen as limited by the foregoing description, butis only limited by the scope of the appended claims.

What is claimed is:
 1. A measurement system, comprising: a measurementplatform having a planar surface; at least two optical sensors coupledto the measurement platform, the optical sensors each having a lightsource, an optical sensor, and a processor that causes light from thelight source to be emitted in a plane and determines a distance to anobject based on a reflection of the light, the at least two opticalsensors being arranged to detect a human operator in a 360 degree areaabout the measurement platform; a linear rail coupled to the measurementplatform; a cooperative robot coupled to move along the linear rail; athree-dimensional (3D) measuring system coupled to the end of thecooperative robot, the 3D measuring system comprising: an imager devicehaving a projector, a first camera and a second camera arranged in apredetermined geometric relationship, the first camera and second cameraeach having a photosensitive array, the projector projecting a patternof light that includes at least one element; one or more firstprocessors operably coupled to the display, the projector, the firstcamera and the second camera, wherein the one or more first processorsare responsive to executable computer instructions for determining adistance to the at least one element; and a controller operably coupledto the at least two optical sensors, the cooperative robot, and the 3Dmeasuring system, the controller having one or more second processorsthat are responsive to executable instructions for changing the speed ofthe robot and the 3D measuring system to be less than a threshold inresponse to a distance measured by at least one of the at least twooptical sensors to a human operator being less than a first distancethreshold.
 2. The system of claim 1, wherein the first distancethreshold is based at least in part on a speed of the cooperative robotand the 3D measuring system.
 3. The system of claim 2, wherein the oneor more second processors are further responsive for reducing the speedof the robot and 3D measuring system to be less than a speed thresholdin response to the distance measured being less than a second threshold.4. The system of claim 3, wherein the speed threshold is equal to orless than 255 mm/second.
 5. The system of claim 3, wherein the firstthreshold defines a first radius about the measurement platform and thesecond threshold defines a second radius about the measurement platform,the second radius being less than the first radius.
 6. The system ofclaim 5, wherein the first radius and second radius are centered on themeasurement platform.
 7. The system of claim 5, wherein the center ofthe first radius and the second radius moves based on a position of the3D measurement device.
 8. The system of claim 5, wherein the center ofthe first radius and the second radius moves based on a position of thecooperative robot.
 9. The system of claim 1, wherein the at least twooptical sensors include a first optical sensor coupled to a first cornerof the measurement platform and a second optical sensor coupled to asecond corner of the measurement platform, the first corner beingdiagonally opposite the first corner.
 10. The system of claim 1, furthercomprising: at least one light coupled to an end of the measurementplatform and operably coupled to the controller, the at least one lightoperable to change between a first color, a second color and a thirdcolor; and wherein the controller is responsive to change the color ofthe at least one light based at least in part on an operational state ofthe cooperative robot.
 11. The system of claim 1, wherein theoperational state of the cooperative robot includes a robot stoppedstate, a robot waiting state, and an operating state.
 12. A method ofmeasuring an object, the method comprising: placing an object on ameasurement platform; moving a cooperative robot at a first speed alonga rail coupled to the measurement platform; scanning the object with athree-dimensional (3D) measuring system coupled to an end of thecooperative robot, the 3D measuring system including an imager devicehaving a projector, a first camera and a second camera arranged in apredetermined geometric relationship, the first camera and second cameraeach having a photosensitive array, the projector projecting a patternof light that includes at least one element; scanning a plane about themeasurement platform with at least one optical sensor; detecting, withthe at least one optical sensor, a human operator at a first distance;and changing the movement of the robot to a second speed when the firstdistance is less than a first distance threshold, the second speed beingless than the first speed.
 13. The method of claim 12, furthercomprising determining the first distance threshold based at least inpart on the first speed.
 14. The method of claim 13, further comprising:detecting, with the at least one optical sensor, that the human operatoris at a second distance; determining a second threshold based at leastin part on the second speed; and changing the movement to a third speedwhen the second distance is less than the second distance threshold, thethird speed being less than the second speed.
 15. The method of claim14, wherein the third speed is less than or equal to 255 mm/second. 16.The method of claim 14, further comprising: determining a first radiusbased on the first threshold; determining a second radius based on thesecond threshold; and wherein the second radius is less than the firstradius.
 17. The method of claim 16, further comprising moving a centerof the first radius and a center of the second radius based at least inpart on a position of the 3D measuring system.
 18. The method of claim16, further comprising moving a center of the first radius and a centerof the second radius based at least in part on a position of thecooperative robot.
 19. The method of claim 12, wherein the at least oneoptical sensor includes a first optical sensor coupled to a first cornerof the measurement platform and a second optical sensor coupled to asecond corner of the measurement platform, the second corner beingdiagonally opposite the first corner.
 20. The method of claim 12,further comprising changing a color of an light coupled to an end of themeasurement platform based at least in part on an operating state of thecooperative robot.